feat: Update Datafusion to v49 #1704
Conversation
DerGut
force-pushed
the
update-datafusion
branch
from
56ac1a4 to
8b07df4
Compare
DerGut
force-pushed
the
update-datafusion
branch
from
8b07df4 to
2ac6349
Compare
There was a problem hiding this comment.
LGTM, thanks for working on the upgrade!
Datafusion 49 introduces a breaking change and is not backwards compatible with older versions.
More context: #1647 (comment)
This means pyiceberg-core will only work with datafusion 49 or newer.
I think we should add merge this into the next release 0.8.0 so that we can align with pyiceberg-core and pyiceberg
|
|
||
| [tool.hatch.envs.dev] | ||
| dependencies = ["maturin>=1.0,<2.0", "pytest>=8.3.2", "datafusion==45.*", "pyiceberg[sql-sqlite,pyarrow]>=0.10.0", "fastavro>=1.11.1"] | ||
| dependencies = ["maturin>=1.0,<2.0", "pytest>=8.3.2", "datafusion==49.*", "pyiceberg[sql-sqlite,pyarrow]>=0.10.0", "fastavro>=1.11.1"] |
There was a problem hiding this comment.
one major side effect is that upgrading to version 49 will make this backwards incompatible. Only other datafusion 49 can work with this.
There was a problem hiding this comment.
That's the painpoint of doing major version upgrading.
There was a problem hiding this comment.
from what i heard, 49->50 is also backwards incompatible
Update datafusion-ffi Regenerate Cargo.lock Update to arrow 55.2 Update to parquet 55.2 Update rust_decimal to 1.37.2 Regenerate Cargo.lock Upgrade datafusion-python to 49
Fix test conflicts
DerGut
force-pushed
the
update-datafusion
branch
2 times, most recently
from
91d7631 to
5046eb8
Compare
3 participants
Which issue does this PR close?
Closes #1702
What changes are included in this PR?
Are these changes tested?